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Abstract. - We present an analytic method of assessing the typical performance of low-density 
parity-check codes on finite-state Markov channels. We show that this problem is similar to a 
spin-glass model on a 'small-world' lattice. We apply our methodology to binary-symmetric and 
binary-asymmetric channels and we provide the critical noise levels for different degrees of channel 
symmetry. 
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Introduction. — A common problem in modern mo- 
bile telecommunication systems is that the strength of the 
ignal varies over time as a result of e.g. the motion of the 
receiver with respect to the source and the varying num- 
ber of obstacles that shadow the signal over time. Chan- 
nels describing communication of attenuated signals are 
termed 'fading channels'. Fading channels are modeled 
by finite-state Markov channels (FSMC) [1]. These chan- 
nels have fueled significant research activity (for a recent 
review on the subject see [2]). In FSMCs there exist a 
number of different channel states that correspond to the 
various possible attenuation factors. Each of the states 
describes a memoryless channel characterized by an error 
probability, while, the transition from one state to an- 
other occurs according to a stationary Markov process. 
Since there are different states in the FMSC the error- 
probabilities between subsequent uses of the channel are 
correlated, i.e. there is memory in the channel. 

One of the central problems in the domain of error- 
correcting codes is the design of codes that reach Shan- 
non's limit. The gap between the Shannon limit and the 
computational limit was closed by turbo codes [3] and by 
low-density parity-check codes (LDPC) [4,5]. For erasure 
channels it was shown that LDPC can reach the Shan- 
non capacity [6] while for general symmetric channels one 
can approach the Shannon limit [7]. To design capacity 
approaching LDPC-codes one uses the density evolution 
(DE) equations to determine the decoding thresholds [8]. 
Since channels with memory have a higher capacity [9,10] 
one would like to introduce memory in the decoding pro- 



E-mail: izaak .neriSfys . kuleuven.be 
( b )E-mail: nikolaos . skantzosOf ortis . com 



cess. Important therefore are the extensions of turbo codes 
and LDPC codes to FSMCs [11, 12]. 

Statistical physics has entered the stage of error correct- 
ing codes after the discovery that the decoding problem 
describing interactions between parity checks and code- 
word variables can be mapped to large frustrated sys- 
tems of interacting particles [13]. Since then, physicists 
have analyzed the performance of Gallager, MacKay-Neal 
and Turbo codes over binary-symmetric, -asymmetric, or 
real- valued channels [14-18] (for a review see [19]). The 
main actor in this approach is the generating function 
of the a posteriori probability distribution of codewords. 
This is similar to the free energy of spin models. Using 
the replica method one derives directly the so-called den- 
sity evolution equations [8] from the free energy. More- 
over the tools of statistical mechanics can be used to cal- 
culate the error-exponents [20,21], MAP-thrcsholds [22] 
and modified schemes of belief-propagation using replica 
symmetry-breaking effects [23]. Generally, the lion's share 
of the volume of research on error-correcting codes has 
been dedicated to memoryless channels. Apart from the 
work of [24], channels with memory, or any other FSMC 
models, have never been to our knowledge analyzed within 
statistical physics. 

Our work is based on techniques that were developed to 
analyze macroscopic properties of 'small-world' networks. 
These systems, due to their close relation with real- world 
networks, have been the subject of intense study from a va- 
riety of scientific disciplines [25-27]. Small- world lattices 
have a particular architecture that allows both a high clus- 
tering coefficient and a small shortest path-length (unlike 
the random Erdos-Renyi graphs). They are constructed 
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by superimposing random and sparse graphs with a finite 
average connectivity onto a one-dimensional ring. An ex- 
act analysis of the thermodynamic properties of such sys- 
tems can be found in [28]. As it is, FSMCs can be mapped 
to small-world lattices, whereby messages between par- 
ity checks and codeword-nodes propagate along the sparse 
graph while messages between channel-state nodes propa- 
gate along the one-dimensional chain. 

In this letter, we present a general method to derive 
the density evolution equations for symmetric or asym- 
metric FSMCs. This includes an exact analysis of the 
Gilbert-Elliot channel (GEC) [29,30]. Fully asymmetric 
cases could be used to describe burst errors in VLSI cir- 
cuits [31,32]. We compute the decoding thresholds for the 
different channels. For symmetric FSMCs we compare the 
results to [12] while for memoryless channels to [18,33]. 

Definitions. — Let us now be more particular. A 
signal er° g { — 1,1}^, prior to its communication over the 
channel, is encoded to er g {-1,1} A/ with M > N. The 
set of codewords C of an LDPC-code is defined by its parity 
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check matrix 
with (H * er), - llj=1 ~. 
(C, -fQ-regular LDPC-codes the parity check matrices are 
random, sparse matrices of dimension (M — N) x M with 
Hjj g {0, 1} and with K non-zero elements per row and 
C non-zero elements per column. 

Channel noise can be modeled with the transformation 
er — > p where the output of the channel p g { — 1,1} M 
depends on the input through the state variable s g S M : 



M 



P(p\s, cr) = JJ ( ) J-Pstate(s) 



(1) 



The probability of the states P s tate(s) follows a Markov 
process 
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We will denote by er°, s° g { — 1, 1} M the true codeword 
and true channel state vectors respectively that were real- 
ized during the signal communication. Depending on the 
definition of the Markov process and the channel noise one 
has different FSMCs. The derivation of the DE equations 
stays mainly the same. We consider two-state Markov- 
modulated binary channels. For these channels the noise 
is a random variable drawn from the distribution 
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Sj = B, (Ti = 1 

Si = B, a l = -1 

Si = G, (7,; = 1 

Si = G, (Ti = -1 



The channel can be in two states: S = {G, B}. Since we 
take {pb + <1b) > {pg + Sg)j B is called the bad state 
and G is called the good state. The Markov process is 




Fig. 1: A graphical representation of the 2-state FSMC where 
in each state the channel is a binary asymmetric channel. The 
bad state B has a higher noise level than the good state G. 



determined by the transition probability W given by 

1-6 b 
9 1 - 9 
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with g the transition probability from state B to G and b 
the transition probability from state G to B (fig. 1). We 
define the memory fn at time step i of the Markov process 
as 



M = W [s e = s\s Q = s]—W[st = s\s Q ^ s] 



(5) 



From (4) we find Hi = (1 — g — b) = // with the time 
index 1 = 1,2, .. . and \x E [—1, 1]. For /i > we have per- 
sistent memory: the probability of remaining in a given 
state is higher than the steady-state probability of being 
in that state. For /i < wc have an oscillatory mem- 
ory. We also define the good-to-bad ratio p = The 
FSMCs we consider are determined by the 6-tuple T = 
(/x, p,pB,qB,PG,Qc)- The GEC [29,30] corresponds to the 
subset of channels Tqec = (a*, P,Pb,Pb,Pg,Pg)- We will 
also consider channels 7as = (/-*, P, nqs, kqg, <Ig) with 
k G [0,1] and Tz = (p, p, 0, q, q, 0). The latter channel 
could be useful for modeling blocks of bad memory or 
bursts of unidirectional noise in VLSI circuits. 

Density Evolution Equations. The starting 
point for the derivation of the DE equations is the cal- 
culation of the generating function / of the a posteriori 
probability distribution of the codeword cr given the chan- 
nel's output p and the parity check matrix H: 
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Using Bayes' law and (1) we obtain 
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with P(er |H) the initial probability distribution of the 
codewords and Z a normalisation constant. We will 
consider unbiased sources of i.i.d.r.v: Pmit(ff) = 2~ M . 
Pcha.n(p\s, er) gives the a priori probability distribution of 
the output p given the state vector s and the codeword er 
(3). The Kronccker delta constrains the summation only 
to those codewords that obey the parity check equation. 

Averaging the generating function over the ensemble 
of parity-check matrices, true-states, true-codewords and 
outputs gives 

-7= lim i- V P(H)P statc ( S °)Pcha„(p|sV°) 

M— >oc iVl — J 
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plus irrelevant constant terms. The probability distribu- 
tion of the parity-check matrices P(H) of a (C, if )-regular 
code can be written in terms of a tensor with K indices 
and elements in {0, 1}, such that the probability that an 
element of the tensor is 1 is C Apr=r and the sum of the el- 
ements equals C for all of its indices (see e.g. [18,19]). The 
free energy / can then be calculated using the replica trick 
(\ogZ) = lim„^ \ log(Z n ). This results, for M -> oo, in 
a saddle point integral. The free energy at the saddle point 
is given by 

-/ = lim -Uxtr p (p(a, a), P(<x, a)) , (8) 

with the exponent of the saddle point integral. The 
extremization is taken over the order parameter functions 
-P(er, er) and P(er , a). These represent the usual order pa- 
rameter functions describing finite connectivity systems, 
see for instance [34], with er = (er 1 , . . . , cr n ) 6 { — 1,1}" 
originating from the replication of the dynamic codeword- 
variables while a 6 { — 1,1} stems from the inclusion of 
the quenched true codeword in the order function. 

The exponent reaches a minimum at the values 

P(er, er), P(er, er) ) that satisfy the saddle point equations: 
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P(er,a) = 

where we defined 
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and we introduced the average (■) p \ s o tT over P c han(p|s°, t). 
Note that while the summations over the replicated code- 
word variables {er;} i=1 N have been performed by reduc- 
ing the graph into a single-site problem, the summations 
over the replicated channel-state variables {si} i=1 N is 
written as a trace over matrix products in (10). This con- 
stitutes the key difficulty in our problem as we arc dealing 
with the (2™ + 1) x (2™ + 1) replicated transfer matrix: 

(s,s°\V(P)\s>, (s )'} = J^W [(s')° \s°] J] W [{s') a \s a ] 

CT .a a 

x (P(er, (H P chan (p| S Q , a a )) p \ s o, a 



To proceed further we now have to make an assumption 
with regards to the structure of the replica space. The 
simplest, replica symmetric ansatz, assumes that 



P(er,cr) = 2-y J dhW{h\a)\\ 
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for some densities W, Z. For the left- and right- 
eigenvectors L(s, s), R(s, s) of V we now assume 

(s',s'\R) = J2W)fdx^ R (x\s')e x ^( s T(U) 

s' 

(L\s,s) = J2 V l(s) I dy<P L (y\s)ey^ sa (15) 

s J 

The form of the above two equations follows the central 
assumption of [28,35]. It allows us to take the remaining 
trace in (10). All distributions above arc normalized at 
n — > 0. The densities Pr and Vl represent respectively 
the right- and left- eigenvectors of W: 



s 'o 
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Following similar computations as in [28,36], we derive in 
the limit n — > the closed, self-consistent equations 
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and also 
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Equations (18-23) are the DE equations for the binary 
asymmetric two-state Markov channel. They describe the 
evolution of the densities of messages propagating along a 
tripartite graph. The graph consists of a chain of channel- 
state nodes connected to codeword nodes and these in turn 
to parity check ones. This graphical representation of the 
decoding process corresponds to an efficient algorithm [38] , 
equivalent to the sum-product algorithm used in channels 
without memory. The tripartite graph has three different 
sets of vertices: the set V co( ie of codeword nodes, the set 
V pc of parity check nodes and the set Khan of channel- 
state nodes, see fig. 2. Due to the presence of memory 
there are 6 types of messages propagating according to: 
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i G Vcode 
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The update equations for single-graph instances for 
these messages (the so-called 'message-passing' equations) 
[38] correspond to the functions within the delta functions 
in the DE equations (18-23). 

Results. — We are interested in deriving the critical 
noise levels beyond which decoding is not possible. This 
information can be obtained through the observable p a = 
Yli£i„ a i = / dhW (h\a)sign(h) where I a describes the 
sublattice I a = {i G Kode|°f = c} and W the distribution 
of the marginals of the decoding variables 



W(h\a) 




~^ du r Z(u r \a) 



channel state 
nodes 




Fig. 2: The tripartite graph and the messages propagating 
along the graph for a LDPC code on channels with memory. 
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The value p a = 1 corresponds to perfect decoding (ferro- 
magnetic phase) while p a < 1 describes decoding failure 
(paramagnetic phase) . We detect the transition by numer- 
ically solving the DE equations (e.g. through population 
dynamics [37]). 

The decoding thresholds in the parameter space (^(pg+ 
Qg),^(pb + 1b)) for a Gallager (C, K) = (3,4) code 
on a 7As-channel are shown in fig. 3. Dotted lines 
separate ferro- from paramagnetic solutions for memory- 
less channels with p = 0, while symbols correspond to 
channels with memory for p = 0.9. We show four de- 
grees of channel asymmetry characterized by the variable 
n = Pb/qb = pg/qg- Note that to simplify the presen- 
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Fig. 3: Decoding thresholds (markers) of a (3, 4)-regular 
LDPC code on a 7As-channel presented in the space of 
\\{vb + Qb), \ {pg + 9g)J for different values of the asymmetry 
K £ [0, 1]. For all symbols the memory equals /i = 0.90 while 
the good-to-bad ratio is p = 1. The dotted line represents the 
memory less threshold /j, = 0. 



Fig. 4: Decoding thresholds in the (fi, q) space of (C, ir- 
regular LDPC codes on a 7z-channel for a good-to-bad ra- 
tio p = 1. The upper dashed lines represent the decoding 
thresholds for the corresponding memoryless binary asymmet- 
ric channel. The lower one corresponds to a value that is twice 
that of the decoding threshold of a memoryless binary sym- 
metric channel. 



tation of our results the two channel states have here the 
same n. The decoding thresholds for n = are computed 
from the DE equations for k — > 0, which can be derived 
when rescaling the fields h — » f3h, u — > f3u, £ — > /3£ and 
£ — » /3£ with /3 = — ilog(ft). The points marked by the 
star-symbols correspond to the points where the two chan- 
nel states have the same error probability, pb — Pg and 
are taken from [18]. Beyond the star-symbol (lower-right 
part of the fig.) the roles of the 'good' versus the 'bad' 
channel are interchanged. In this fig. we also see that 
both the presence of memory and that of asymmetry in 
the channel allows for higher noise levels. In the limiting 
cases of k = 1 our results agree very well with those of [12] . 
In table 1 we give the decoding thresholds corresponding 
to fig. 3. 

In fig. 4 we present results from the channel 7z in 
which there exist two Z-type states: (jpb,c[b) = (0, (?) 
and (pg:9g) = (^O) (hence the terms 'good' vs 'bad' are 
not very meaningful here) . This type of configuration can 
model 'burst-error' channels where a very large number 
of consecutive bits appear corrupted while the corruption 
is selective with regards to the input symbol. We show 
results in the (/j,, q) space for Gallager (C, K) = (3, 4) and 
(4,6) codes. The lower dashed line corresponds to the 
noise level 2<7bsc where <7bsc is the critical level of the 
memoryless binary-symmetric channel. The fact that the 
marker at fj, = coincides with the dashed line is not a 



coincidence since in this limit the channel has two com- 
plementary Z-type states without memory, and therefore, 
with an equal transition probability between them. The 
upper dashed line corresponds to the critical noise level of 
a memoryless Z-channel [18]. At fj, = — 1 the transition 
probabilities become b = g = 1 and thus the channel os- 
cillates between the two states. We note that this fig. is 
symmetric with respect to the fi = axis; a property that 
also follows from the DE equations. 

Conclusions. — Error-correcting codes on channels 
with memory are known to outperform the traditional 
ones on memoryless channels. They can be used in mod- 
ern mobile communication systems or to model burst-error 
channels. In this letter we have presented a technique for 
deriving the density evolution equations for multi-state 
channels. This method is based on the diagonalisation of 
replicated transfer matrices that was originally developed 
to study 'small-world' systems. It turns out that the rep- 
resentation of the LDPC multi-state decoding problem on 
graphs shares a common architecture with 'small-world' 
systems: In memoryless channels, decoding occurs with 
message-passing between symbol variables (the 'spins') 
which are connected to parity-check variables (the 'cou- 
plings'). Channels with memory introduce a new element 
to this hypergraph which can be seen and treated as a 
chain of channel-state variables with nearest-neighbor in- 
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K = 1 


K = 0.1 


K = 0.01 


K = 


(PG + 9g)/2 




(pb4 


9b)/2 




0.20 
0.08 
0.05 


/ 

0.262(2) 
0.308(2) 


0.190(2) 
0.331(2) 
0.396(2) 


0.242(2) 
0.401(2) 
0.484(2) 


0.256(2) 
0.420(2) 

/ 



Table 1: Critical noise levels for the (3,4)-regular LDPC code 
on a TAs-channel with memory fx = 0.9, good-to-bad ratio 
p = 1 and for four degrees of channel asymmetry. 



teractions. 

We have presented results for the Gilbert-Elliott chan- 
nel and its generalisation to asymmetric two-state chan- 
nels with memory. The density evolution equations that 
follow from the analysis reproduce very well the special 
limiting cases of the GEC or the memoryless binary- 
asymmetric channel. The method can be applied to 
a variety of multi-state error-correcting codes, such as 
multi-symbol, gaussian-, non-Markovian or intersymbol- 
interference channels. From a statistical physics point of 
view an interesting future direction would be the inclusion 
of replica symmetry-breaking effects [39] which might cor- 
rect the critical noise levels we present here. 
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